🐛 Workload should still resilient when catalog is deleted #2439

camilamacedo86 · 2026-01-11T05:18:07Z

Problem

When a catalog becomes unavailable (deleted, registry offline, network issues), installed extensions break or stop being maintained. This PR ensures extensions continue working with their installed version until the catalog becomes available again.

What This Fixes

Issues on main when catalog is unavailable/deleted:

Accidentally deleted resources are NOT restored (both runtimes)
Configuration changes are blocked (both runtimes)
Auto-updates hang during catalog image transitions (both runtimes)
Status shows "Failed" instead of "Retrying" (both runtimes)
Version upgrade attempts show unclear status (both runtimes)
Catalog deletion/maintenance breaks extensions (both runtimes)
Resources drift if manually modified (Helm only)
Registry outages break extension health (Helm only)

Note: Boxcutter already maintains resources via CER controller; Helm did not.

Solution

Added smart fallback logic:

Catalogs exist but resolution fails → Retry immediately (transient issue)
Catalogs deleted → Fall back to installed bundle, continue maintaining resources
Version change requested → Always retry (cannot upgrade without catalog)

Key Changes

Resolution step - Added catalog existence check before fallback
Helm applier - Added reconcileExistingRelease() to maintain resources when contentFS == nil
Boxcutter applier - Return success when contentFS == nil (CER controller maintains)
Status - Clear "Retrying" status instead of "Failed"

What "Extension Continue Working" Means

An extension continues working when:

Operator Deployment is running and healthy
All managed resources exist in cluster
Deleted resources are automatically restored
Resource specs match desired state (no drift)
Status shows Installed=True
Operator's business logic continues (e.g., Prometheus keeps scraping)

Testing

Added comprehensive e2e test suite in test/e2e/features/catalog-deletion-resilience.feature:

Extension continues running after catalog deletion
Resources auto-restored after catalog deletion
Config changes work without catalog
Version upgrades correctly blocked without catalog
Multiple revisions remain stable (Boxcutter)
Workload availability properly tracked

All scenarios tested for both Helm and Boxcutter runtimes where applicable.

What Still Requires Catalog (Correct Behavior)

Fresh installs
Version upgrades
Package changes

Resolution Fails?

├─ Version change requested (1.0.0 → 1.0.1)?
│  └─ YES → RETRY (need catalog to upgrade)
│
├─ Catalogs exist in cluster?
│  ├─ YES → RETRY (transient issue, catalog updating)
│  └─ NO → Check for installed bundle...
│     ├─ Have installed bundle?
│     │  └─ YES → FALLBACK (maintain current workload)
│     └─ NO → RETRY (fresh install needs catalog)

TL;DR Reconcile Workflow and Scenarios

Step 1: Resolution + rollout succeed (healthy)

RollingOut: empty
Resolution runs → succeeds
Rollout completes → moves to Installed
State: Installed = v1.0.0, RollingOut = []

Step 2: Resolution succeeds, rollout starts, rollout fails partway

RollingOut: empty initially
Resolution runs → succeeds (v2.0.0)
Start rolling out v2.0.0
Rollout FAILS partway through (e.g., resource can't be applied)
State: Installed = v1.0.0, RollingOut = [v2.0.0] ← v2.0.0 stuck here

Step 3: Catalog missing; resolution would fail, but we skip it

What happens:

if len(state.revisionStates.RollingOut) == 0 {  // FALSE (we have v2.0.0 rolling out)
    // Skip resolution entirely
} else {
    // Use the rolling-out revision
    resolvedRevisionMetadata = state.revisionStates.RollingOut[0]  // v2.0.0
}

Result:

We CONTINUE trying to roll out v2.0.0
We skip resolution entirely (don’t check if catalog exists)
We don’t fall back to v1.0.0
We keep retrying the v2.0.0 rollout that already started
We don’t need the catalog because v2.0.0 content was already obtained when resolution succeeded in Step 2

Why this makes sense:

Already committed: v2.0.0 was resolved while the catalog was available
Have the content: bundle image for v2.0.0 was already pulled/cached
Finish what we started: failure is applying resources, not resolving
No catalog needed: retrying rollout doesn’t require catalog access

When does fallback happen?
Fallback to Installed only happens when:

RollingOut is empty
AND resolution fails
AND catalog doesn't exist
AND no version change was requested

In this scenario RollingOut is populated, so fallback never triggers.

/hold until we have a RFC approved

RFC: Catalog Deletion Resilience for ClusterExtensions
Internal discussion conveys: https://redhat-internal.slack.com/archives/C06KP34REFJ/p1768291605594589?thread_ts=1768235428.251779&cid=C06KP34REFJ

Closes: #209

netlify · 2026-01-11T05:18:11Z

✅ Deploy Preview for olmv1 ready!

Name	Link
🔨 Latest commit	`d1759f0`
🔍 Latest deploy log	https://app.netlify.com/projects/olmv1/deploys/696eca02582883000867a763
😎 Deploy Preview	https://deploy-preview-2439--olmv1.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

Copilot

Pull request overview

This PR adds comprehensive end-to-end tests to verify that installed OLM extensions continue functioning correctly when their source catalog is deleted. The tests cover both standard runtime and experimental Boxcutter runtime scenarios.

Changes:

Added new feature file with 8 scenarios testing catalog deletion resilience
Implemented CatalogIsDeleted function to support catalog deletion in tests
Added step registrations for ClusterExtension update operations

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File	Description
test/e2e/steps/steps.go	Adds CatalogIsDeleted function and step registrations for testing catalog deletion and ClusterExtension updates
test/e2e/features/catalog-deletion-resilience.feature	Defines 8 test scenarios covering extension resilience, resource restoration, config changes, version upgrades, and revision behavior when catalog is deleted

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

test/e2e/steps/steps.go

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

test/e2e/steps/steps.go

internal/operator-controller/applier/boxcutter.go

internal/operator-controller/controllers/clusterextension_reconcile_steps.go

internal/operator-controller/applier/helm.go

Copilot

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

test/e2e/features/catalog-deletion-resilience.feature

internal/operator-controller/applier/boxcutter.go

Copilot

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 5 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

internal/operator-controller/applier/boxcutter.go

internal/operator-controller/controllers/clusterextension_reconcile_steps.go

internal/operator-controller/applier/helm.go

internal/operator-controller/controllers/clusterextension_reconcile_steps.go

test/e2e/steps/steps.go

camilamacedo86 · 2026-01-18T05:06:54Z

Hi @pedjak @joelanford

All feedbacks are addressed.
Also, I enhance a little the code, logs and error messages.

Please, feel free to check it out.

Copilot

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated 3 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

internal/operator-controller/controllers/clusterextension_reconcile_steps.go

camilamacedo86 · 2026-01-19T09:32:19Z

Hi @tmshort

This one should either close the one that you open a long time ago: #209

perdasilva · 2026-01-19T12:49:42Z

I wouldn't have expected any code changes to the Appliers, only to the resolution step since we already have the RetrieveRevisionStates step. I would have expected that everything would live in the Resolution step which would choose to keep what is currently installed. Then, everything downstream from resolution stays basically the same. No?

test/e2e/features/recover.feature

Copilot

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated 3 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

internal/operator-controller/controllers/clusterextension_reconcile_steps.go

Copilot

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated 3 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

test/e2e/features/recover.feature

internal/operator-controller/controllers/clusterextension_admission_test.go

camilamacedo86 · 2026-01-19T17:31:05Z

Hi @perdasilva

Thx for looking that.

I wouldn't have expected any code changes to the Appliers, only to the resolution step since we already have the RetrieveRevisionStates step. I would have expected that everything would live in the Resolution step which would choose to keep what is currently installed. Then, everything downstream from resolution stays basically the same. No?

Resolution can decide "keep what’s installed", but if the catalog/registry is gone we can’t unpack/render the bundle content anymore. So, the changes in the Applier are required because contentFS == nil check. If the applier always assumes contentFS exists, it will error/crash and reconciliation stops (We cannot unpack). For Helm, we still want reconciliation to keep healing drift (restore deleted resources), so Helm needs a fallback path when contentFS is nil.

Copilot

Pull request overview

Copilot reviewed 15 out of 15 changed files in this pull request and generated 2 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

internal/operator-controller/applier/helm.go

test/e2e/steps/hooks.go

tmshort · 2026-01-19T21:21:59Z

/approve

Approved to go in, but there are still updates that need to happen

openshift-ci · 2026-01-19T21:22:06Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: tmshort

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [tmshort]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

camilamacedo86 · 2026-01-20T00:17:48Z

Hi @pedjak
All addressed. I think we are good now for a new round.

@tmshort all addressed now.
If you wish give a second look please feel free.

Thank you a lot for all your time and amazing help 🙌

Assisted-by: Cursor

Enables installed extensions to continue working when their source catalog becomes unavailable or is deleted. When resolution fails due to catalog unavailability, the operator now continues reconciling with the currently installed bundle instead of failing. Changes: - Resolution falls back to installed bundle when catalog unavailable - Unpacking skipped when maintaining current installed state - Helm and Boxcutter appliers handle nil contentFS gracefully - Version upgrades properly blocked without catalog access This ensures workloads remain stable and operational even when the catalog they were installed from is temporarily unavailable or deleted, while appropriately preventing version changes that require catalog access. Assisted-by: Cursor

Copilot AI review requested due to automatic review settings January 11, 2026 05:18

openshift-ci bot requested review from ankitathomas and trgeiger January 11, 2026 05:18

Copilot started reviewing on behalf of camilamacedo86 January 11, 2026 05:18 View session

Copilot AI reviewed Jan 11, 2026

View reviewed changes

test/e2e/steps/steps.go Outdated Show resolved Hide resolved

test/e2e/steps/steps.go Outdated Show resolved Hide resolved

camilamacedo86 force-pushed the test-e2e-res branch from d3cbb5a to f31b184 Compare January 11, 2026 05:43

Copilot AI review requested due to automatic review settings January 11, 2026 05:43

Copilot started reviewing on behalf of camilamacedo86 January 11, 2026 05:43 View session

camilamacedo86 changed the title ~~🌱 test: add e2e tests for workload resilience when catalog is deleted~~ WIP 🌱 test: add e2e tests for workload resilience when catalog is deleted Jan 11, 2026

openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jan 11, 2026

Copilot AI reviewed Jan 11, 2026

View reviewed changes

test/e2e/steps/steps.go Outdated Show resolved Hide resolved

test/e2e/steps/steps.go Outdated Show resolved Hide resolved

camilamacedo86 force-pushed the test-e2e-res branch from f31b184 to dce6d68 Compare January 11, 2026 06:48

camilamacedo86 commented Jan 11, 2026

View reviewed changes

internal/operator-controller/applier/boxcutter.go Outdated Show resolved Hide resolved

camilamacedo86 commented Jan 11, 2026

View reviewed changes

internal/operator-controller/controllers/clusterextension_reconcile_steps.go Show resolved Hide resolved

camilamacedo86 commented Jan 11, 2026

View reviewed changes

internal/operator-controller/applier/helm.go Show resolved Hide resolved

camilamacedo86 force-pushed the test-e2e-res branch from dce6d68 to b15c262 Compare January 11, 2026 07:09

Copilot AI review requested due to automatic review settings January 11, 2026 07:09

Copilot started reviewing on behalf of camilamacedo86 January 11, 2026 07:10 View session

Copilot AI reviewed Jan 11, 2026

View reviewed changes

test/e2e/features/catalog-deletion-resilience.feature Outdated Show resolved Hide resolved

internal/operator-controller/applier/boxcutter.go Outdated Show resolved Hide resolved

camilamacedo86 force-pushed the test-e2e-res branch from b15c262 to b1d259e Compare January 11, 2026 07:26

Copilot AI review requested due to automatic review settings January 11, 2026 07:30

camilamacedo86 force-pushed the test-e2e-res branch from b1d259e to c6870c5 Compare January 11, 2026 07:30

Copilot started reviewing on behalf of camilamacedo86 January 11, 2026 07:30 View session

camilamacedo86 changed the title ~~WIP 🌱 test: add e2e tests for workload resilience when catalog is deleted~~ WIP 🐛 Workload should still resilient when catalog is deleted Jan 11, 2026

Copilot AI reviewed Jan 11, 2026

View reviewed changes

camilamacedo86 force-pushed the test-e2e-res branch from c6870c5 to 36e9069 Compare January 11, 2026 08:48

Copilot AI review requested due to automatic review settings January 11, 2026 09:00

camilamacedo86 force-pushed the test-e2e-res branch from 36e9069 to 6799025 Compare January 11, 2026 09:00

Copilot started reviewing on behalf of camilamacedo86 January 11, 2026 09:01 View session

Copilot started reviewing on behalf of camilamacedo86 January 18, 2026 05:05 View session

camilamacedo86 changed the title ~~WIP 🐛 Workload should still resilient when catalog is deleted~~ 🐛 Workload should still resilient when catalog is deleted Jan 18, 2026

openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jan 18, 2026

Copilot AI reviewed Jan 18, 2026

View reviewed changes

camilamacedo86 force-pushed the test-e2e-res branch from 008a58e to 7c23624 Compare January 18, 2026 05:38

pedjak reviewed Jan 19, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings January 19, 2026 16:26

Copilot started reviewing on behalf of camilamacedo86 January 19, 2026 16:26 View session

Copilot AI reviewed Jan 19, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings January 19, 2026 16:54

Copilot started reviewing on behalf of camilamacedo86 January 19, 2026 16:54 View session

Copilot AI reviewed Jan 19, 2026

View reviewed changes

test/e2e/features/recover.feature Outdated Show resolved Hide resolved

test/e2e/features/recover.feature Outdated Show resolved Hide resolved

internal/operator-controller/controllers/clusterextension_admission_test.go Show resolved Hide resolved

camilamacedo86 force-pushed the test-e2e-res branch from e6fb69f to c554d3d Compare January 19, 2026 17:13

camilamacedo86 force-pushed the test-e2e-res branch from c554d3d to 7264c2a Compare January 19, 2026 19:27

Copilot AI review requested due to automatic review settings January 19, 2026 19:27

Copilot started reviewing on behalf of camilamacedo86 January 19, 2026 19:27 View session

Copilot AI reviewed Jan 19, 2026

View reviewed changes

internal/operator-controller/applier/helm.go Show resolved Hide resolved

test/e2e/steps/hooks.go Outdated Show resolved Hide resolved

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jan 19, 2026

camilamacedo86 force-pushed the test-e2e-res branch from 7264c2a to 669fed0 Compare January 20, 2026 00:15

camilamacedo86 requested a review from pedjak January 20, 2026 00:18

camilamacedo86 added 2 commits January 20, 2026 00:19

test: add e2e tests for workload resilience when catalog is deleted

cc41210

Assisted-by: Cursor

camilamacedo86 force-pushed the test-e2e-res branch from 669fed0 to d1759f0 Compare January 20, 2026 00:19

🐛 Workload should still resilient when catalog is deleted #2439

Are you sure you want to change the base?

🐛 Workload should still resilient when catalog is deleted #2439

Conversation

camilamacedo86 commented Jan 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

What This Fixes

Solution

Key Changes

What "Extension Continue Working" Means

Testing

What Still Requires Catalog (Correct Behavior)

Resolution Fails?

TL;DR Reconcile Workflow and Scenarios

Step 1: Resolution + rollout succeed (healthy)

Step 2: Resolution succeeds, rollout starts, rollout fails partway

Step 3: Catalog missing; resolution would fail, but we skip it

Uh oh!

netlify bot commented Jan 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for olmv1 ready!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

camilamacedo86 commented Jan 18, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

camilamacedo86 commented Jan 19, 2026

Uh oh!

perdasilva commented Jan 19, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

camilamacedo86 commented Jan 11, 2026 •

edited

Loading

netlify bot commented Jan 11, 2026 •

edited

Loading